Facial Gestures: Taxonomy and Application of Non-Verbal, Non-Emotional Facial Displays for Embodied Conversational Agents

Facial displays are an extremely important communication channel fulfilling a wide variety of functions in discourse and conversation. Humans use them naturally, often subconsciously, and are therefore very sensitive to the application of such displays in their computer-generated correspondents, Embodied Conversational Agents (ECA). In this chapter, we aim to provide an extensive survey of one class of facial displays, the facial gestures. Facial gestures include various nods and head movements, blinks, eyebrow gestures and gaze, i.e., all facial displays except explicit verbal and emotional displays (visemes or expressions such as smile). Consciously or subconsciously, facial gestures play an important role both in discourse and in conversation. They are instrumental in turn taking, emphasizing, providing rhythm, and can be connected to physiological functions. While verbal and emotional displays may be regarded as the explicit, perhaps even obvious, the facial gestures are less tangible yet they are largely responsible for what we intuitively call natural behavior of the face. In other words, an ECA pronouncing a sentence using perfect coarticulation mechanism for the lips and displaying a carefully modeled expression of surprise will still look unnatural if the facial gestures are not right as well. It is therefore extremely important for an ECA to implement facial gestures well. While there is a large body of knowledge on this topic both from psychology and ECA literature, it is quite scattered. Existing ECA implementations typically concentrate on some aspects of facial gestures but do not cover the complete set. We attempt to provide a complete survey of facial gestures that can be useful as a guideline for their implementation in an ECA. Specifically, we provide a systematically organized repertoire of usual facial gestures. For each gesture class, we provide the information on its typical usage in discourse and conversation, conscious or subconscious causes, any available knowledge on Conversational Informatics: An Engineering Approach Edited by Toyoaki Nishida © 2007 John Wiley & Sons, Ltd typical dynamics and amplitude of the gesture. Finally we provide an example of a practical system implementation as a case study.